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LIFE  CYCLE  COSTING  OF  SIMULATED  VS  ACTUAL  EQUIPMENT 
FOR  INTERMEDIATE  MAINTENANCE  TRAINING 

F.  Thomas  Eggemeier  and  Gary  A.  Klein 
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Air  Force  Human  Resources  Laboratory 
Wright-Patterson  Air  Force  Base  Ohio  A5433 

Life  cycle  cost  estimates  of  training  equipment  for  F-16  Avionics  Intermediate 

Station  personnel  were  developed.  The  major  purpose  was  to  compare  the  cost 

of  intermediate  level  maintenance  training  when  conducted  on  simulated  vs 

actual  avionics  test  equipment.  This  was  the  initial  phase  of  a  planned 

two-part  effort.  The  analysis  was  therefore  limited  to  estimates  of  training 

device  acquisition  and  maintenance  costs.  Total  estimated  fifteen-year  costs 

for  simulated  equipment  trainers  were  approximately  5056  less  than  comparable 

estimates  for  actual  equipment  trainers. 

INTRODUCTION 

This  paper  reports  the  initial  results  of  a  two-phase  effort  to  develop 
life  cycle  cost  (LCC)  estimates  of  training  equipment  for  F-16  Avionics 
Intermediate  Station  (AIS)  maintenance  personnel.  The  goal  of  the  entire 
effort  is  to  estimate  the  LCC  of  simulated  test  equipment  versus  actual  test 
equipment  in  training  personnel  to  operate  and  maintain  the  F-16  AIS.  This 
initial  phase  was  a  preliminary  analysis  of  major  cost  factors  differentiating 
simulated  and  actual  test  equipment.  It  was  conducted  to  provide  an  early 
estimate  of  the  cost  of  a  training  simulator  and  also  to  decide  if  a  more 
detailed  LCC  study  was  warranted. 


The  AIS  is  the  set  of  maintenance  equipment  required  to  test  aircraft 
avionics  units  which  have  malfunctioned  on  the  aircraft  and  have  therefore 
been  removed.  The  AIS  is  used  to  identify  the  specific  fault  in  the  unit  so 
that  the  operator  can  remove  and  replace  the  subcomponent  responsible  for  the 
malfunction.  The  operator  is  responsible  for  using  the  AIS  to  test, 
troubleshoot,  and  perform  maintenance  actions  on  units.  Maintainer  personnel, 
on  the  other  hand,  are  responsible  for  maintaining  and  repairing  the  AIS 
itself.  In  this  context,  the  AIS  itself  constitutes  one  training  device  which 
could  be  used  to  support  the  training  of  operators  and  maintainer  personnel, 
and  a  simulated  AIS  would  constitute  an  alternative  training  device. 

Simulators  have  a  number  of  potential  advantages  over  actual  equipment  in 
maintenance  training,  primarily  in  the  areas  of  cost,  safety,  and  reliability 
(Miller  &  Rockway,  1975;  Modrick,  Kanarick,  Daniels,  &  Gardner,  1975).  For 
example,  since  actual  equipment  trainers  normally  operate  with  high  voltages, 
instructors  are  often  reluctant  to  allow  students  to  perform  hands-on 
training,  fearing  injury  to  the  student  or  damage  to  the  equipment. 
Consequently,  the  effectiveness  of  the  trainer  is  compromised.  Several  other 
factors  contribute  to  the  potential  for  increased  effectiveness  of  simulators 
relative  to  actual  equipment.  These  include  the  ability  of  the  simulator  to 
train  the  repair  of  a  broader  rangfe  of  malfunctions,  permit  greater 
environmental  control,  and  provide  a  wider  variety  of  instructional  actions 
and  conditions.  Ease  of  device  modification  and  greater  reliability  are  also 
potential  advantages  of  simulation  over  actual  equipment.  These  potential 
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advantages  make  simulation  an  alternative  worthy  of  serious  consideration  when 
acquiring  training  equipment. 

On  the  other  hand,  the  use  of  actual  AIS  equipment  for  training  has  some 
potential  advantages  when  compared  to  a  simulated  equipment  trainer.  The 
actual  equipment  is  more  realistic,  thereby  potentially  promoting  greater  user 
acceptance.  It  is  readily  available  as  an  add-on  procurement.  Simulators 
require  special  procurements,  the  developmeht  of  new  hardware  and  software, 
and  tne  introduction  of  new  maintenance  requirements  into  the  supply  system. 

Among  the  potential  differences  between  simulators  and  actual  equipment, 
cost  is  obviously  of  major  importance  in  deciding  which  type  of  training 
equipment  to  procure.  This  paper  addresses  the  cost  aspect  of  choice  of  a 
training  device. 


METHOD 

Cost  Model 

Cost  factors  for  the  analysis  were  selected  from  a  training  cost  model 
developed  by  Braby,  Henry,  Parrish,  and  Swope  (1975).  The  model  includes  the 
provision  for  costing  a  variety  of  training  program  factors.  The  model 
includes  six  major  areas:  facilities,  equipment,  instructional  material, 
personnel,  supplies,  and  students. 

The  Braby  et  al.  (1975)  model  has  been  previously  used  to  estimate  costs 
of  intermediate  maintenance  simulators  (Daniels  &  Cronin,  1975;  Daniels, 
1976).  Both  previous  efforts  demonstrated  a  significant  potential  for  cost 
savings  with  use  of  simulated  equipment.  Both  studies  also  indicated  that 
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equipment  costs  and  school  personnel  costs  were  the  two  major  areas  which  were 
significantly  affected  by  use  of  simulators  rather  than  actual  equipment.  In 
both  studies,  these  two  categories  accounted  for  over  95*  of  the  estimated  LCC 
savings  with  use  of  simulated  equipment.  Of  the  two  factors,  equipment  costs 
represented  the  most  significant  element  differentiating  actual  from  simulated 
equipment  and  accounted  for  more  than  60*  of  the  estimated  savings. 

Since  equipment  costs  represent  the  majority  of  estimated  savings 
attributed  to  simulation  in  previous  analyses,  it  was  decided  that  this 
preliminary  estimate  would  be  conducted  with  primary  focus  on  equipment 
costs.  Emphasis  in  this  effort  was  therefore  placed  on  the  cost  factor  with 
the  greatest  potential  to  show  a  clear  differentiation  between  simulation  and 
actual  equipment  trainers. 

Procedure 

A  design  team,  consisting  of  training  psychologists  and  a  simulation 
engineer,  was  assembled  to  perform  the  analysis.  To  meet  the  objective  of 
comparing  simulated  and  actual  equipment  costs,  the  design  team  applied  a 
three-step  process.  During  Step  One,  analyses  were  conducted  to  determine  a 
set  of  training  requirements  for  test  station  operators  and  maintainers.  Step 
Two  consisted  of  developing  a  conceptual  design  for  a  representative  AIS 
simulation  that  would  satisfy  those  training  requirements.  During  Step  Three, 
acquisition  and  maintenance  costs  were  derived  for  the  training  simulator  and 
for  an  actual  equipment  trainer. 
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Derivation  of  Training  Requirements 

Operator  Training  Requirements.  Training  requirements  should,  ideally,  he 
derived  from  a  complete  Instructional  Systems  Development  (ISD)  analysis. 
However,  ISD  data  were  not  available  for  use  during  this  effort.  Therefore, 
it  was  necessary  to  identify  operator  training  requirements  orimarily  through 
an  analysis  of  task  data  provided  in  an  F-16  Task  and  Skill  Analysis  Report. 
Tnis  report  was  supplemented  with  information  gathered  through  interviews  witn 
F-16  personnel  and  F-15  AIS  instructor  and  maintenance  personnel.  A  technioue 
of  analyzing  analogous  training  situations  to  derive  the  training  requirements 
of  new  systems  (Cream,  Eqgemeier,  &  Klein,  1978)  was  used  in  this  effort. 
Cnoice  of  the  F-15  AIS  was  based  on  the  fact  that  it  was  the  closest  existing 
analog  to  the  planned  F-16  AIS. 

An  F-16  AIS  consists  of  four  test  stations.  Each  station  is  automatic,  in 
that  its  software  is  capable  of  performing  the  majority  of  necessary  tests 
without  operator  intervention,  once  the  aircraft  avionics  unit  has  been 
connected  to  the  station.  Avionics  units  are  connected  to  the  test  station  by 
means  of  interface  test  adaptors.  The  automatic  testing  procedures  are 
designed  to  identify  malfunctions  to  the  subcomponent  level.  The  subcomponent 
can  then  be  removed  and  replaced  by  the  operator. 

The  task  analyses  and  interview  data  indicated  that  in  general,  the 
primary  functions  of  the  F-16  AIS  operator  are  to  set  up  the  test  station, 
perform  the  necessary  procedures  to  test  the  units,  remove  and  replace 
defective  subcomponents,  and  verify  proper  operation  of  the  avionics  units 
after  they  nad  been  repaired. 
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Task  data  were  gathered  for  each  avionics  unit  at  each  test  staticni  and 
summarized  in  a  matrix  format.  From  the  matrix,  it  was  possible  to  identify 
patterns  of  operator  tasks  and  relate  these  to  avionics  units.  The  matrix  was 
then  used  to  determine  the  minimum  number  of  units  which  would  ensure  training 
of  each  of  the  tasks  required  to  operate  the  test  station.  On  the  basis  of 
this  procedure,  twelve  of  thirty-four  avionics  units  were  identified  as 
training  requirements.  Only  the  peripheral  equipment  and  the  interface  test 
adaptors  required  to  test  these  selected  units  were  included  as  training 
requirements. 

Maintainer  Training  Requirements.  Derivation  of  training  requirements  for 
maintainer  personnel  was  more  complex  than  for  the  operator,  due  to  the 
absence  of  any  maintainer  task  data.  During  interviews,  F-15  AIS  and  F-16 
engineering  prsonnel  directly  identified  areas  within  the  test  station  that 
would  be  expected  to  fail  most  frequently  and  be  most  difficult  to  repair. 
While  this  procedure  was  not  as  thorough  as  listing  all  tasks,  it  provided  a 
direct  assessment  of  the  anticipated  major  training  requirements. 

It  was  determined  that  the  major  requirement  for  the  station  maintainer  is 
access  to  the  interior  of  the  test  station  in  order  to  effect  repairs. 
Interviews  pinpointed  the  computational  and  switching  element  within  a  test 
station  as  the  areas  of  least  reliability.  It  was  therefore  decided  that  the 
repair  of  the  Computer  and  the  Switching  Unit/Measurement  and  Stimulus  System 
(SUMSS)  elements  in  the  F-16  AIS  should  be  among  the  primary  training 
requirements. 
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Development  of  Simulation  Requirements 

The  methods  used  to  derive  necessary  simulation  capability  from  training 
requirements  have  been  described  by  Cream,  Eggemeier,  and  Klein  (1975,  1978). 
In  the  case  of  the  operator  tasks,  the  derivation  was  straightforward.  A  set 
of  representative  avionics  units,  test  adaptors,  and  peripheral  equipment  had 
been  identified  as  necessary  to  provide  adequate  training.  The  simulation 
requirements  consisted  of  providing  these  elements  of  equipment  and  the 
components  of  the  test  station  itself  that  were  involved  in  testing  avionics 
units. 

It  was  determined  that  simulation  of  only  front  panels  on  each  test 
station  was  required  for  the  operator,  who  typically  works  with  only  the 
exterior  portions  of  the  station.  Three  representations  of  front  panels  were 
used:  (1)  actual  equipment;  (2)  simulated  panels  with  knobs,  switches,  and 
displays  functional;  and  (3)  mock-up  via  photo  or  other  type  of  flat 
representation.  In  order  to  provide  a  direct  LCC  comparison  with  actual 
equipment,  a  simulator  configuration  which  included  all  four  test  stations  was 
developed. 

Determination  of  simulator  capabilities  for  maintainer  personnel  was  also 
based  upon  the  primary  training  requirements  that  had  been  identified. 

Training  requirements  were  developed  in  terms  of  the  electronic  cards  within 
the  test  station.  The  determination  of  simulator  capabilities  indicated  that 
the  trainee  should  be  able  to  perform  troubleshooting  actions  down  to  the 
level  of  testing  cards  and  components  of  cards  within  the  test  station. 
Simulation  of  a  core  set  of  SUMSS  cards  and  a  number  of  computer  sub-assembly 


card  modules  was  therefore  necessary  to  provide  the  maintainer  with 
trouDleshooting  experience  on  each  category  of  card  function  within  the 
station. 

The  frequency  of  use  of  each  of  the  SUMSS  cards  was  determined  from  the 
F-16  AIS  Preliminary  Design  Review  data,  as  was  the  circuit  function, 
connection  in  the  test  station,  and  distribution  throughout  the  test  station. 
Cards  were  then  selected  so  that  a  representative  of  each  major  functional 
category  was  included  in  the  simulation.  On  the  basis  of  that  criterion,  29 
cards  distributed  across  the  four  test  stations  were  chosen  for  simulation. 

The  remaining  316  cards  were  designated  for  mock-up  only. 

Pertinent  literature  was  assembled  and  analyzed  to  determine  computer 
Chassis  access  procedures,  computer  circuit  functions,  and  computer  card 
complexity.  Card  function  was  the  criterion  used  to  assess  the  appropriate 
degree  of  simulation  for  the  computer.  Subassembly  computer  card  modules  were 
chosen  for  simulation  to  represent  a  broad  range  of  functions.  On  the  basis 
of  these  criteria,  ten  computer  card  modules  were  chosen  for  some  degree  of 
simulation.  Simulated  operating  components,  simulated  components,  and  photo 
or  other  flat  representation  mock-ups  comprised  the  major  simulation 
categories. 


RESULTS 

Simulator  Acquisition  Cost  Estimate 

Estimates  of  acquisition  costs  for  each  simulated  test  station  were 
developed.  Acquisition  cost  estimates  were  based  on  three  major  cost 
factors:  (1)  Design  and  fabrication  of  simulators;  (2)  Engineering  change 
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proposals  during  the  simulator  development  cycle;  and  (3)  Technical  Order  data 
required  to  support  the  simulators  when  they  had  been  placed  in  the  field. 

Design  and  fabrication  costs  included  estimated  design  and  software 
development  costs,  costs  for  fabrication  of  simulated  test  stations,  and  the 
cost  of  the  materials  required  for  the  fabrication.  Cost  estimates  in  each 
case  were  based  upon  "best  commercial  practices."  The  total  estimated  design 
and  fabrication  cost  for  the  simulated  test  stations  was  $3.3  million  (M) 
dollars. 

In  order  to  estimate  the  costs  of  engineering  change  proposals,  it  was 
assumed  that  a  four-year  development  cycle  would  be  required  to  procure  the 
simulated  stations.  Engineering  change  costs  were  then  estimated  in  the 
following  manner:  20%  of  the  estimated  desigh  and  fabrication  costs  for  the 
first  year  of  development,  and  15%,  10%,  and  5%  of  design  and  fabrication 
costs  for  the  second,  third,  and  fourth  years  of  development,  respectively. 

Tne  total  estimated  engineering  change  cost  for  the  simulated  stations  was 

$1.6M. 

Costs  associated  with  procurement  of  technical  data  for  the  simulators 
were  estimated  to  be  7.25%  of  the  estimated  design  and  fabrication  costs. 
Estimated  technical  data  costs  were  $250  thousand  (K)  dollars. 

The  estimated  total  acquisition  cost  for  the  simulated  stations  is 
therefore  $5.2M. 

Simulator  Maintenance  Cost  Estimate 

Estimates  of  fifteen-year  maintenance  costs  for  the  simulator  were 
developed.  It  was  assumed  that  major  maintenance  work  on  the  simulator  would 
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be  contracted.  The  manufacturer  of  the  computer  specified  to  drive  the 
simulation  provided  an  estimate  of  labor  for  one  year  on  the  computer  and 
associated  peripherals.  Although  it  was  estimated  that  the  remaining 
electronics  in  the  simulated  test  station  would  not  equal  the  computer  in 
complexity  to  maintain,  the  labor  estimate  for  the  computer  was  doubled  to 
arrive  at  a  conservative  one-year  labor  estimate  for  the  entire  simulated  test 
station.  The  one-year  labor  estimate  was  inflated  at  10%  per  year  for  the 
fifteen  years  of  the  life  cycle  in  order  to  arrive  at  the  fifteen-year 
estimate  of  maintenance  labor  for  one  simulated  test  station.  The  estimate 
for  one  station  was  then  multiplied  by  four  to  arrive  at  an  estimate  for  all 
four  stations.  Estimated  fifteen-year  maintenance  labor  costs  were  $432K. 

In  order  to  estimate  the  costs  of  spares  for  the  simulation,  a  percentage 
of  the  total  design  and  fabrication  costs  for  the  four  simulated  test  stations 
was  calculated.  This  percentage  varied  over  the  fifteen-year  cycle,  beginning 
with  approximately  9%  of  design  and  fabrication  costs  per  year  for  the  first 
two  years,  and  1%  of  design  and  fabrication  costs  per  year  for  the  remaining 
thirteen  years  of  the  life  cycle.  It  was  assumed  that  the  majority  of 
simulator  spares  would  consist  of  initial  spares  lay-in  with  minimal 
requirements  for  replenishment  spares.  This  assumption  is  based  on  the 
assumed  nigh  reliability  of  simulators.  Each  year's  estimates  were  then 
summed  to  yield  the  fifteen-year  estimate  for  spares  for  all  four  stations. 

The  estimated  fifteen-year  cost  of  spares  was  $722K. 

The  total  estimate  fifteen-year  maintenance  costs  for  the  simulated  AIS  is 
therefore  $1.15M. 
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Total  Cost  Estimate  for  the  Simulator 


A  summary  of  the  major  cost  elements  for  the  simulated  AIS  is  provided  in 
Table  1.  The  total  fifteen-year  cost  estimate  is  $6.35M. 

TABLE  1 

Summary  of  Major  Cost  Elements  for  the  Simulated  AIS 
Estimated  Acquisition  Cost  $5.2CIM 

Estimated  15- Year  Maintenance  Cost  $1.1 5M 

Total  Estimated  Simulated  AIS  Cost _ $6.35M 

Estimated  Actual  Equipment  Costs 

Data  were  available  from  Air  Force  training  device  procurement  sources  on 
the  estimated  costs  for  acquisition  of  all  four  actual  test  stations,  a 
complete  set  of  thirty-four  avionics  units  and  interface  test  adaptors, 
calibration  requirements,  and  the  costs  of  initial  spares  for  two  years  for 
the  test  stations  and  associated  peripheral  equipment.  Since  the  analysis  of 
station  operator  functions  conducted  to  specify  simulation  requirements  had 
indicated  that  only  twelve  of  the  thirty-four  avionics  units  and  associated 
test  adaptors  would  be  required  to  accomplish  the  required  training,  the 
actual  equipment  costs  estimates  were  reduced  35*.  This  reduction  was 
accomplished  in  order  to  make  the  configuration  of  actual  equipment  as 
comparable  as  possible  to  that  of  the  simulator.  The  adjusted  total 
acquisition  cost  for  the  actual  AIS  was  $7.15M. 

Since  the  available  data  provided  estimates  for  the  initial  two  years  of 
spares,  the  remaining  thirteen  years  of  spares  were  calculated  at  1*  of 
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acquisition  costs  per  year.  The  estimated  fifteen-year  cost  of  snares  for  the 
actual  AIS  was  $2.25M. 

Data  were  also  available  from  Air  Force  sources  concerning  the  estimated 
cost  of  Air  Force  manpower  required  to  maintain  the  actual  AIS  for  fifteen 
years.  The  total  estimated  fifteen-year  labor  cost  for  the  actual  AIS  was 
$3.3M. 

The  total  estimated  fifteen-year  maintenance  cost  for  the  actual  AIS  is 
therefore  $5.55M. 

A  summary  of  major  cost  elements  for  the  actual  AIS  is  orovided  in  Table 
2.  The  total  fifteen-year  cost  estimate  is  $12. 7M 

TABLE  2 


Summary  of  Major  Cost  Elements  for  the  Actual  AIS 


Estimated  Acquisition  Cost 


$7.]5M 


Estimated  15- Year  Maintenance  Cost 


$5.55M 


Total  Estimated  Actual  AIS  Cost 


$12. 7M 


DISCUSSION 

The  results  indicate  that  the  procurement  and  maintenance  of  simulation 
offers  a  significant  potential  for  cost  avoidance  when  compared  with  actual 
equipment  costs.  Total  estimated  fifteen-year  costs  for  simulated  equipment 
were  $6.35M,  while  comparable  estimates  for  actual  equipment  were  $12. 7M,  a 
savings  of  50%  with  use  of  simulated  equipment.  Efforts  were  made  to  provide 
a  direct  comparison  of  actual  and  simulated  equipment  by  considering  all  four 
test  stations,  specifying  equal  numbers  of  avionics  units  for  both  actual 
equipment  and  the  simulation,  and  excluding  automated  instructional  features 
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from  consideration.  It  has  been  suggested  (e.g.,  Daniels  \  Drnnin, 

Daniels,  1976)  that  simulation  permits  automation  of  instructional  functions 
tnat  can  permit  significant  savings  in  instructor  costs,  and,  possibly, 
trainee  costs.  This  effort  did  not  address  tnis  potential  cost  impact  for  the 
reasons  cited  earlier. 

It  should  be  noted  that  this  analysis  addressed  only  the  cost  aspect  of 
choice  of  training  device.  The  question  of  the  relative  training 
effectiveness  of  simulation  and  actual  equipment  *ias  beyond  the  scope  of  this 
effort.  At  the  present  time,  there  are  no  reports  in  tne  literature  *rtiich 
compare  the  training  effectiveness  of  simulators  and  actual  equipment  for 
intermediate  maintenance  training.  Such  training  effectiveness  data,  '«hen 
available,  will  obviously  be  of  critical  importance  in  choosing  a  training 
device. 

Based  on  the  present  results,  plans  have  been  made  to  initiate  the 
detailed  LCC  analysis.  The  detailed  LCC  study  will  assess  the  adequacy  c'  the 
techniques  used  in  this  study,  with  the  goal  of  determining  how  they  might  be 
refined  for  future  efforts.  Also,  the  detailed  analysis  can  indicate  the 
general  level  of  confidence  that  can  be  expected  with  the  methods  used  to 
develop  the  estimate  in  this  report. 

Since  this  study  was  undertaken  at  a  time  when  the  actual  F-16  AIS  was  in 
a  state  of  flux,  some  current  estimates  may  require  alteration,  due  to 
decisions  made  subsequent  to  the  analysis  period.  Attempts  have  been  made  to 
minimize  error  by  using  the  latest  available  data. 


The  potential  value  of  this  effort  extends  beyond  determininq  whether  or 
not  to  continue  with  a  subsequent  detailed  LCC  study.  It  demonstrates  a 
method  for  deriving  cost  estimates  based  on  limited  data.  No  actual  F-16  AIS 
stations  were  in  existence  during  this  work.  Further,  there  were  no 
simulators  in  existence  during  this  work  that  were  of  this  type  of  training 
device,  although  two  such  simulators  were  under  construction.  Therefore,  the 
configuration  of  an  appropriate  simulation  had  to  be  predicted,  since  there 
were  only  minimal  task  analysis  data  available  for  the  nonexistent  AIS 
station.  The  requirement  for  LCC  estimation  in  the  absence  of  complete  data 
is  not  atypical,  and  this  report  describes  methods  for  providing  such 
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